Empirical Methods for Evaluating Dialog Systems
نویسنده
چکیده
We examine what purpose a dialog metric serves and then propose empirical methods for evaluating systems that meet that purpose. The methods include a protocol for conducting a wizard-of-oz experiment and a basic set of descriptive statistics for substantiating performance claims using the data collected from the experiment as an ideal benchmark or “gold standard” for comparative judgments. The methods also provide a practical means of optimizing the system through component analysis and cost valuation. Empirical Methods for Evaluating Dialog Systems
منابع مشابه
Visualizing Empirical Dialog Trajectories
Automated spoken dialog systems require systematic procedures for evaluating performance and diagnosing problems. We present an interactive tool that provides graphical views of how users are navigating and interacting with the system. The technology analyzes all calls, providing fine-grained analysis and diagnosis, for system evaluation and business intelligence. The input is a continuous feed...
متن کاملAn Integrated Dialog Simulation Technique for Evaluating Spoken Dialog Systems
This paper proposes a novel integrated dialog simulation technique for evaluating spoken dialog systems. Many techniques for simulating users and errors have been proposed for use in improving and evaluating spoken dialog systems, but most of them are not easily applied to various dialog systems or domains because some are limited to specific domains or others require heuristic rules. In this p...
متن کاملInteractive visualization of human-machine dialogs
Automated spoken dialog systems require systematic procedures for evaluating performance and diagnosing problems. We present an interactive tool that provides graphical views of how callers navigate through such systems, enabling fine-grained analysis for system evaluation and business intelligence. The input is a feed of call-logs. The output is an empirical dialog trajectory analysis represen...
متن کاملAre We There Yet? Research in Commercial Spoken Dialog Systems
In this paper we discuss the recent evolution of spoken dialog systems in commercial deployments. Yet based on a simple finite state machine design paradigm, dialog systems reached today a higher level of complexity. The availability of massive amounts of data during deployment led to the development of continuous optimization strategy pushing the design and development of spoken dialog applica...
متن کاملEvaluating responsiveness in spoken dialog systems
Ratings of user satisfaction, although fairly easy to elicit for today’s spoken language systems, can be more elusive for systems which operate at near-human levels of performance. This problem can be alleviated by adding a ‘relistening’ phase before eliciting judgements: in this phase the user listens to a recording of himself interacting with the system while consulting a transcript of that i...
متن کامل